Lifecycle management of automated testing

ABSTRACT

Systems and methods for lifecycle management of automated testing are disclosed. In one embodiment, a method includes processing multiple manual test cases for an application under test, associating a set of reusable test scripts to the manual test cases, where the set of reusable test scripts is selected from a library of reusable test scripts, and executing the set of reusable test scripts for the application under test using an automated testing tool associated with the set of reusable test scripts.

This application is a reissue application of U.S. Pat. No. 8,347,147,issued on Jan. 1, 2013, from U.S. patent application Ser. No.12/399,982, filed on Mar. 9, 2009, all of which are incorporated hereinby reference in their entirety.

FIELD OF TECHNOLOGY

Embodiments of the present invention relate to the field of automatedtesting. More particularly, embodiments of the present invention relateto lifecycle management of automated testing.

BACKGROUND

Software needs to be tested before it is delivered. For example, severalfeatures of an application under test (AUT) may be manually executed,and the results may be compared with their expected outcomes. Althoughthe application can be tested manually, the recent trend is movingtoward automated testing. Commonly, automated testing involvesautomating a manual process already in place that uses a formalizedtesting process. Record and playback features of conventional automatedtesting tools (e.g., HP QTP®, IBM RFT®, etc.) may capture and record theuser's actions or manual testing steps to generated test scripts, whichmay be used later for automatic testing of the application under test.

However, as an application changes to adopt new features orfunctionalities, the test scripts may need to be updated or rewrittenfrom scratch, thus ensuing in high maintenance costs. As for themaintenance of the test scripts, it has become harder to acquire testautomation experts due to growing demands and increasing complexities inapplications in general. Furthermore, as test automation projects for anorganization or company spread across different geographical locations,business units, and/or various domains, it has become even moredifficult to manage the test automation projects using the conventionalautomated testing tools.

SUMMARY

Systems and methods of lifecycle management of automated testing aredisclosed. In one aspect, a method for lifecycle management of automatedtesting comprises processing multiple manual test cases for anapplication under test, associating a set of reusable test scripts tothe manual test cases, where the set of reusable test scripts isselected from a library of reusable test scripts, and executing the setof reusable test scripts for the application under test using anautomated testing tool associated with the set of reusable test scripts.

In another aspect, a method for lifecycle management of automatedtesting comprises presenting a guideline to generate multiple manualtest cases for an application under test, accessing a library ofreusable test scripts for an automated testing tool to select a set ofreusable test scripts which correspond to the manual test cases from thelibrary when the automated testing tool is selected from a number oflicensed automated testing tools, and setting respective parameters forthe set of reusable test scripts, and executing the set of reusable testscripts for the application under test using the automated testing tool.

The methods disclosed herein may be implemented in any means forachieving various aspects, and may be executed in a form of a machinereadable medium embodying a set of instructions that, when executed by amachine, cause the machine to perform any of the operations disclosedherein. Other features will be apparent from the accompanying drawingsand from the detailed description that follows.

BRIEF DESCRIPTION OF THE DRAWINGS

Example embodiments are illustrated by way of example and not limitationin the figures of the accompanying drawings, in which like referencesindicate similar elements and in which:

FIG. 1 is a block diagram which illustrates an exemplary lifecyclemanagement of automated testing, according to one embodiment.

FIG. 2 is an exemplary graphical user interface of the requirementmodule of FIG. 1, according to one embodiment.

FIG. 3 is an exemplary graphical user interface for the analysis moduleof FIG. 1, according to one embodiment.

FIG. 4 illustrates exemplary libraries of reusable test scriptsassociated with the design module of FIG. 1, according to oneembodiment.

FIG. 5 illustrates an exemplary process for creating new test cases,according to one embodiment.

FIG. 6 illustrates an exemplary graphical user interface for thedevelopment module of FIG. 1, according to one embodiment.

FIG. 7 illustrates an exemplary view of a return on investment (ROI)report, according to one embodiment.

FIG. 8 illustrates an exemplary lifecycle management system forautomated testing, according to one embodiment.

FIG. 9 is a process flow chart of an exemplary method for lifecyclemanagement of automated testing, according to one embodiment.

FIG. 10 is a process flow chart of another exemplary method forlifecycle management of automated testing, according to one embodiment.

Other features of the present embodiments will be apparent from theaccompanying drawings and from the detailed description that follows.

DETAILED DESCRIPTION

Systems and methods of lifecycle management of automated testing aredisclosed. Embodiments of the present invention include test automationlifecycle management features and a guided engineering technique at eachstage of test automation using reusable test scripts. Thus, the systemsand methods maximize return on investment, ensure cross projectreusability, empower non-automation experts to create automation testsuites, ensure uniformity in automation approach across theorganization, and/or provide a set of guidelines and best practices forthe test automation.

In the following detailed description of the embodiments of theinvention, reference is made to the accompanying drawings that form apart hereof, and in which are shown by way of illustration specificembodiments in which the invention may be practiced. These embodimentsare described in sufficient detail to enable those skilled in the art topractice the invention, and it is to be understood that otherembodiments may be utilized and that changes may be made withoutdeparting from the scope of the present invention. The followingdetailed description is, therefore, not to be taken in a limiting sense,and the scope of the present invention is defined only by the appendedclaims.

FIG. 1 illustrates an exemplary system 100 for lifecycle management ofautomated testing, according to one embodiment. In FIG. 1, the system100 comprises an administration module 102, a report module 104, aproject management module 106, reusable test scripts 108, a requirementmodule 110, an analysis module 112, a design module 114, a developmentmodule 116, and an execution module 118.

The administration module 102 may be used to perform administrativefunctions. For example, it can be used to maintain records foremployees, control the scheduling of automated testing and resources,and/or produce management and operational reports. The report module 104may be used to report the quality of ongoing or completed automatedtesting projects and/or the qualities of application under tests (AUT).The project management module 106 may be used to centrally allocateavailable resources (e.g., experts, licensed automated testing tools,etc.) to one or more automated testing projects and manage theirprogresses. The reusable test scripts 108 may include component codeswhich can be readily used or need some modifications to create anautomated test scenario of multiple test cases. It is appreciated that atest case describes a test that needs to be run on the application undertest to verify that the application under test runs as expected.

The system 100 also comes with in-built guidance at every stage ofautomation which is realized by the requirement module 110, the analysismodule 112, the design module 114, the development module 116, and theexecution module 118. The guideline may offer the best practices of testautomation. The requirement module 110 may be used to capture varioustypes of technical requirements needed for test automation. The analysismodule 112 may be used to analyze a user's responses to the technicalrequirements from the requirement module 110. The design module 114 maybe used to maintain reusable test scripts used for automated testing.The development module 116 may be used to build a test scenario byassociating some of the reusable test scripts 108 to test cases used tobuild the test scenario. The execution module 118 may be used to executethe test scenario using a licensed automated testing tool (e.g., HPQTP®, IBM RFT®, etc.).

In one embodiment, a method for lifecycle management of automatedtesting comprises presenting a guideline across the automated testingmodules which includes the requirement module 110, the analysis module112, the design module 114, the development module 116, and theexecution module 118, to generate multiple manual test cases for anapplication under test. The method also comprises accessing a library ofreusable test scripts for an automated testing tool to select a set ofreusable test scripts which correspond to the manual test cases when theautomated testing tool is selected from a number of licensed automatedtesting tools. The method further comprises setting respectiveparameters for the set of reusable test scripts, and executing the setof reusable test scripts for the application under test using theautomated testing tool.

FIG. 2 is an exemplary graphical user interface (GUI) 200 of therequirement module 110 of FIG. 1, according to one embodiment. The GUI200 may be used to capture test automation requirements of anapplication under test. In one exemplary implementation, the requirementmodule 110 may solicit a user's input to technical requirements 202 ofthe application under test. The technical requirements 202 may comprisean environment 204, a standard 206, a time/resource 208, an integration210, and/or others 212.

One or more questionnaire items in the environment 204 solicit theuser's responses on the environment (e.g., platform support requirements214, OS/browser/application type requirements 216, lingual supportrequirements 218, etc.) in which the test is expected to run. Thestandard 206 solicits the user's responses on the requirements formaintenance (e.g., maintain from multiple places) and execution (e.g.,execute locally or from a remote location). The time/resources 208 maysolicit the user's responses on available licenses, skill levels ofpeople, and/or organization of the test automation project. Theintegration 210 may solicit the user's responses on the project'sintegration with various entities. It is appreciated that there could memore items (e.g., others 212) on the technical requirements. As will beillustrated in detail in FIG. 5, a test case generation 220 may be usedto generate new manual test cases or import from existing manual testcases.

FIG. 3 is an exemplary graphical user interface (GUI) 300 of theanalysis module 112 of FIG. 1, according to one embodiment. The GUI 300may be used to analyze the user's responses to the requirement module110 of FIG. 1. This is done by presenting a pre-built check list ofautomation analysis parameters that enables creation of a suitabledesign, development, and execution strategy for the automation project.As illustrated in FIG. 3, the GUI 300 may present a menu which includesa general 302, analyze test cases 304, analyze integration requirements306, analyze application 308, analyze time/resources 310, and/or analyzereporting/execution-error log requirements 312.

FIG. 4 illustrates exemplary libraries 400 of reusable test scriptsassociated with the design module 114 of FIG. 1, according to oneembodiment. The design module 114 may be used to create the librarieswhere the components or reusable test transcripts may be created usingone or more automating tools (HP QTP®, IBM RFT®, etc.). Then, thecontents of the entire libraries 400 can be exported into a single file,or the components may be exported into individual files. When thecomponents or reusable test scripts are uploaded to the system 100 ofFIG. 1, their codes may be checked for errors. In an alternativeembodiment, as the case base or script comprising functions is uploadedinto the system 100, the libraries or the reusability design may becreated from the script.

In one exemplary implementation, the libraries 400 may be in multiplelevels or layers. For example, the libraries 400 may comprise fivelevels—core framework functions 404, standard window functions 406,windows specific functions 408, entity functions 410, and applicationlevel functions 412. As illustrated in FIG. 4, the application levelsfunctions 412, the entity functions 410, the windows specific functions408, the standard window functions 406, and the core framework functions404 include reusable components or test scripts that depend on eachother. Thus, manageusers 414 of the application level functions 412includes createuser 416 of the entity functions 410. In addition, thecreateuser 416 of the entity functions 410 includes login 418 and user420 of the windows specific functions 408. Then, the login 418 of thewindows specific functions 408 includes login_invoke 422,login_setvalues 424, and login_accept 426. Also, the login_invoke 422,the login_setvalues 424, and the login_accept 426 of the standard windowfunctions 406 include web_launchURL 428, web_settext 430 and 432, andweb_click 434, respectively. Furthermore, the web_launchURL 428 of thestandard windows functions 406 depend on web_launchURL 436, theweb_settext 430 and 432 to web_settext 438, and the web_click 434 toweb_click 440 of the core framework functions 404.

With the layered structure of the libraries 400, the user may identify alayer or level where modification to a particular reusable test scriptneeds to be made. For instance, if more fundamental changes to testscripts need to be made, reusable test scripts at the core frame workfunctions 404 may be modified. Additionally, the reusable test scriptsmay be divided into two sections, one that does not require any GUIapplication (e.g., business flow testing operations) and the other thatrequire one or more GUI applications (e.g., functions that interact withGUI). This design approach may minimize changes or modifications to thereusable test scripts for automated testing projects, thus reducing themaintenance efforts and costs of the testing.

FIG. 5 illustrates an exemplary process 500 for creating new test cases,according to one embodiment. In FIG. 5, a test case manager 502 may be agraphical user interface for creating one or more test cases (e.g.,manual test cases). A test repository 504 may be used to store the testcases. From a drop down menu, the user may click “create new test cases506. Then, a dialog window may be displayed requiring the user's input,such as name 508 (e.g., “login”), ID 510 (e.g., “tc001”), details 512(e.g., “test case to check login functionality”, expected behaviour 514(e.g., “valid user should be able to login”), complexity 516 (e.g.,“simple”), priority 518 (e.g., “medium”), and criticality 520 (e.g.,“important”). Once the test case is created, it is stored in a folder ofthe test repository 504. In an alternative embodiment, one or more testcases may be imported from existing test cases.

FIG. 6 illustrates an exemplary graphical user interface (GUI) 600 forthe development module 116 of FIG. 1, according to one embodiment. TheGUI 600 is used to create a test scenario 602 based on multiple testcases 604. Each of the test cases 604 may include several test steps606. Then each step 606 may be matched with a reusable test script. Oncethe process is completed, the test scenario 602 may be ready forexecution.

In FIG. 6, the test scenario 602, “notepad_type_close,” includes twotest cases, TC1 and TC2. TC1 is based on two test steps 606 with stepnumbers 608 TS1 and TS2, whereas TC2 is based on six test steps withstep numbers 608 TS1-TS6. Each of the steps is described with a stepdescription 610, and its expected result 612 is listed as well.Furthermore, a keyword 614 is used to match a particular reusable testscript to a test case (e.g., or its test step). Then, param 1 616 andparam 2 618 may be used to set parameter values. Additionally, an edittest step dialog 620 may be used to edit one or more of the test steps606. In the dialog box, TS1 of TC2 is used to associate reusable testscript “web_invoke” from the core framework functions of the libraries400 using a select library 622 and a select function 626 menus. Then,parameter values 624 associated with the test step may be set.

FIG. 7 illustrates an exemplary view of a return on investment (ROI)report 714, according to one embodiment. In FIG. 7, a ROI report request702 may be used to process a user's request for the ROI report 714. Forthe request 702, the user is request to fill out the user's organizationunit 704, manual effort billing rate 706 associated with the testautomation project, start date 708, end date 710, and e-mail address712. Then, the ROI report 714 is forwarded to the user in response tothe ROI report request 702 to inform the user about the usage of theautomated testing scripts using different automated testing tools, wherethe report includes a project name 716, month 718, effort saved in hours720, manual testing rate 722, amount saved 724, amount invested 726, andreturn on investment in percentage 728.

FIG. 8 illustrates an exemplary lifecycle management system 800 forautomated testing, according to one embodiment. The lifecycle managementsystem 800 allows the distribution of tasks on the basis of expertise ofits people 806. Thus, design of the automation workflow (e.g., viaactivities 808 which include study & analyze requirements 814,architect/design solution 818, script functions 822, develop test cases830, maintain & execute 834, and report 840) may be assigned moremethodically.

For example, a chief information office (CIO) 802 may appoint managers804 (e.g., automation manager 1—division A, automation manager2—division B. automation manager 3—division C) for multiple testautomation tasks. Each manager then assigns tasks for the project to oneor more of the people 806 according to their expertise. In FIG. 8, thetask of the study & analyze requirements 814 is assigned to anautomation consultant 816 (e.g., using the requirement module 110 andthe analysis module 112 of FIG. 1), the architect/design solution 818 toan automation architect 820 (e.g., using the design module 114), and thescript functions 822 to automation engineers 824 (e.g., using the designmodule 114 & tools 810). Once the script functions 822 are performedusing the tools 810, code 812 may be generated. As illustrated in FIG.8, scripts/test cases/frameworks 828A may be generated by processing thescript functions 822 using automation scripting tool 1 826A,scripts/test cases/frameworks 828B using automation scripting tool 2826B, and scripts/test cases/frameworks 828C using automation scriptingtool 3 826C.

Then, the task of the develop test cases 830 is assigned to a manualtester/business user 832, and a manual tester 836 may be assigned toperform the task of the maintain & execute 834 using an automationscripting tool 838. The same tester 836 or another tester 842 may beassigned to perform a task of the report 840. Thus, the lifecyclemanagement system 800 may define ongoing automation projects in a singleconsole in a hierarchical tree view. In addition, it may provide astandard list of automation tasks that can be assigned, tracked, orreassigned by project managers for single or multiple projects.Furthermore, the lifecycle management system 800 may encompass afacility to assign specific users to certain tasks, and define theirrole requirements and privileges. Moreover, the lifecycle managementsystem 800 may have in-built reports for test automation status andapplication quality reports which can be viewed any location.

FIG. 9 is a process flow chart 900 of an exemplary method for lifecyclemanagement of automated testing, according to one embodiment. Inoperation 902, multiple manual test cases for an application under testare processed. In one embodiment, a guideline for generating themultiple test cases may be presented during the operation, where theguideline may include a list of technical requirements for testing theapplication under test and a list of questionnaires for analyzing a userresponse to the list of technical requirements. The manual test casesmay be imported from existing manual test cases, or they may be newlycreated.

In operation 904, a set of reusable test scripts is associated to themanual test cases, where the set of reusable test scripts is selectedfrom a library of reusable test scripts. In one embodiment, the libraryof reusable test scripts for the automated testing tool may be accessedwhen the automated testing tool is selected from a number of licensedautomated testing tools. In addition, respective parameters for the setof reusable test scripts may be set. Furthermore, the library ofreusable test scripts may be in multiple levels for easy maintenance ofthe reusable test scripts, where the multiple levels may include a fieldlevel, a window level, a module level, and an application functionalitylevel.

In operation 906, the set of reusable test scripts (e.g., or a testscenario) for the application under test is executed using an automatedtesting tool associated with the set of reusable test script, if the setof reusable test scripts at the field level is generated using theautomated testing tool.

It is appreciated that the methods disclosed herein may be executed in aform of a machine readable medium embodying a set of instructions that,when executed by a machine, cause the machine to perform any of theoperations disclosed herein.

FIG. 10 is a process flow chart 1000 of another exemplary method forlifecycle management of automated testing, according to one embodiment.In operation 1002, a guideline is presented to generate multiple manualtest cases for an application under test. In operation 1004, a libraryof reusable test scripts for an automated testing tool is accessed toselect a set of reusable test scripts which correspond to the manualtest cases when the automated testing tool is selected from a number oflicensed automated testing tools. In operation 1006, respectiveparameters for the set of reusable test scripts are set. In operation1008, the set of reusable test scripts is executed for the applicationunder test using the automated testing tool.

It is appreciated that the methods disclosed herein may be executed in aform of a machine readable medium embodying a set of instructions that,when executed by a machine, cause the machine to perform any of theoperations disclosed herein.

Although the present embodiments have been described with reference tospecific example embodiments, it will be evident that variousmodifications and changes may be made to these embodiments withoutdeparting from the broader spirit and scope of the various embodiments.For example, the various devices, modules, analyzers, generators, etc.described herein may be enabled and operated using hardware circuitry(e.g., CMOS based logic circuitry), firmware, software and/or anycombination of hardware, firmware, and/or software (e.g., embodied in amachine readable medium). For example, the various electrical structureand methods may be embodied using transistors, logic gates, andelectrical circuits (e.g., application specific integrated ASICcircuitry).

What is claimed is:
 1. A method for lifecycle management of automatedtesting, comprising: processing a plurality of manual test cases for anapplication under test; associating a set of reusable test scripts tothe plurality of manual test cases, wherein the set of reusable testscripts is selected from a library of reusable test scripts, wherein thelibrary of reusable test scripts is accessed for an automated testingtool when the automated testing tool is selected from a number oflicensed automated testing tools; executing the set of reusable testscripts for the application under test using the automated testing toolassociated with the set of reusable test scripts; displaying automatedtesting projects which include the automated testing of the applicationunder test; and displaying a return on investment (ROI) for each of theautomated testing projects.
 2. The method of claim 1, wherein theprocessing the plurality of manual test cases comprises presenting aguideline for generating the plurality of manual test cases.
 3. Themethod of claim 2, wherein the guideline comprises a list of technicalrequirements for testing the application under test.
 4. The method ofclaim 3, wherein the guideline comprises a list of questionnaires foranalyzing a user response to the list of technical requirements.
 5. Themethod of claim 1, wherein the associating the set of reusable testscripts further comprises setting respective parameters for the set ofreusable test scripts.
 6. The method of claim 1, wherein at least one ofthe plurality of manual test cases is imported from existing manual testcases.
 7. The method of claim 1, wherein at least one of the pluralityof manual test cases is newly created.
 8. The method of claim 1, whereinthe automated testing projects which include the automated testing ofthe application under test are displayed in a hierarchical tree view. 9.The method of claim 8, wherein the lifecycle management of the automatedtesting comprises displaying a standard list of automation taskscontrolled by at least one project manager for the automated testingprojects.
 10. The method of claim 8, wherein the lifecycle management ofthe automated testing comprises defining specific users and their rolerequirements and privileges for the automated testing projects.
 11. Themethod of claim 8, wherein the lifecycle management of the automatedtesting comprises forwarding a status report for each automated testingproject and a quality report for each application under test.
 12. Amethod for lifecycle management of automated testing, comprising:processing a plurality of manual test cases for an application undertest; associating a set of reusable test scripts to the plurality ofmanual test cases, wherein the set of reusable test scripts is selectedfrom a library of reusable test scripts, wherein the library of reusabletest scripts is in multiple levels for easy maintenance of the reusabletest scripts; and executing the set of reusable test scripts for theapplication under test using an automated testing tool associated withthe set of reusable test scripts.
 13. The method of claim 12, whereinthe multiple levels comprise a field level, a window level, a modulelevel, and an application functionality level.
 14. The method of claim13, wherein each reusable test script at the field level is generatedusing the automated testing tool.
 15. A method for lifecycle managementof automated testing, comprising: processing a plurality of manual testcases for an application under test; associating a set of reusable testscripts to the plurality of manual test cases, wherein the set ofreusable test scripts is selected from a library of reusable testscripts; executing the set of reusable test scripts for the applicationunder test using an automated testing tool associated with the set ofreusable test scripts; displaying automated testing projects whichinclude the automated testing of the application under test in ahierarchical tree view; and displaying a return on investment (ROI) forthe each automated testing project.
 16. A method for lifecyclemanagement of automated testing, comprising: presenting a guideline togenerate a plurality of manual test cases for an application under test;accessing a library of reusable test scripts for an automated testingtool to select a set of reusable test scripts which correspond to theplurality of manual test cases when the automated testing tool isselected from a number of licensed automated testing tools, wherein thelibrary of reusable test scripts is in multiple levels for easymaintenance of the reusable test scripts; setting respective parametersfor the set of reusable test scripts; and executing the set of reusabletest scripts for the application under test using the automated testingtool.
 17. The method of claim 16, wherein at least one of the pluralityof manual test cases is newly created or imported from existing manualtest cases.
 18. A non-transitory computer readable medium for lifecyclemanagement of automated testing having instructions that, when executedby a computer, cause the computer to perform a method comprising:processing a plurality of manual test cases for an application undertest; associating a set of reusable test scripts to the plurality ofmanual test cases, wherein the set of reusable test scripts is selectedfrom a library of reusable test scripts, wherein the library of reusabletest scripts is in multiple levels for easy maintenance of the reusabletest scripts; and executing the set of reusable test scripts for theapplication under test using the automated testing tool associated withthe set of reusable test scripts.
 19. A system embodied on anon-transitory computer-readable storage medium for lifecycle managementof automated testing, the system comprising: an administration modulefor performing administrative functions; a report module for reportingon quality of automated testing projects and application under tests(AUT); a project management module for centrally allocating availableresources to the automated testing projects and managing progress of theautomated testing projects; a set of reusable test scripts comprisingcomponent codes used to create an automated test scenario of a pluralityof manual test cases, wherein the set of reusable test scripts isselected from a library of reusable test scripts, and wherein thelibrary of reusable test scripts is in multiple levels for easymaintenance of the reusable test scripts; a requirement module forcapturing technical requirements needed for the automated testing; ananalysis module for analyzing a user's responses to the technicalrequirements from the requirement module; a design module formaintaining the set of reusable test scripts used for the automatedtesting; a development module for building a test scenario for anapplication under test by associating the set of reusable test scriptsto the plurality of manual test cases; and an execution module forexecuting the test scenario for the application under test using alicensed automated testing tool associated with the set of reusable testscripts.
 20. The method of claim 1, wherein displaying a return oninvestment (ROI) for each of the automated testing projects comprises:displaying the return on investment (ROI) for a plurality of theautomated testing projects in a single report.
 21. The method of claim20, wherein the return on investment (ROI) for the plurality of theautomated testing projects are displayed via a graphical user interfaceon a display screen.
 22. The method of claim 21, wherein the pluralityof manual test cases for the application under test are determined usinga graphical user interface user input.
 23. The method of claim 15,wherein displaying a return on investment (ROI) for each of theautomated testing projects comprises: displaying the return oninvestment (ROI) for a plurality of the automated testing projects in asingle report.
 24. The method of claim 23, wherein the return oninvestment (ROI) for the plurality of the automated testing projects aredisplayed via a graphical user interface on a display screen.
 25. Themethod of claim 24, wherein the plurality of manual test cases for theapplication under test are determined using a graphical user interfaceuser input.